Method for requesting and viewing a zoomed area of detail from an image attachment on a mobile communication device

ABSTRACT

A process is set forth for viewing an enlarged area of an image. The image is stored on a server and re-sized for viewing on a mobile communication device based on screen size and color display capabilities of the device. The image is enlarged within the server by modifying binary raw data of the original image based on crop rectangle coordinates entered at the mobile communication device. The process allows users to quickly retrieve any relevant part of a large image attachment that has been resized by the server. This minimizes bandwidth usage, device memory/CPU consumption, and request/response latency while still allowing the user to view an image area in its original level of detail.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.12/199,230, filed Aug. 27, 2008 and entitled “Method for Requesting andViewing a Zoomed Area of Detail form an Image Attachment on a MobileCommunication Device,” which is a continuation of U.S. Pat. No.7,451,405 issued Nov. 11, 2008 (application Ser. No. 10/941,763, filedSep. 15, 2004), and entitled “Method of Requesting and Viewing a ZoomedArea of Detail form an Image Attachment on a Mobile CommunicationDevice,” each of which is expressly incorporated by reference herein inits entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The following is directed in general to displaying content on mobilecommunication devices, and more particularly to a method for viewing anarea of detail from an image attachment on a mobile communicationdevice.

2. Brief Description of the Prior Art

Mobile communication devices are becoming increasingly popular forbusiness and personal use due to a relatively recent increase in numberof services and features that the devices and mobile infrastructuressupport. Handheld mobile communication devices, sometimes referred to asmobile stations, are essentially portable computers having wirelesscapability, and come in various forms. These include Personal DigitalAssistants (PDAs), cellular phones and smart phones. While their reducedsize is an advantage to portability, limitations in download bandwidthand device screen size give rise to challenges in viewing large imageshaving high horizontal and vertical resolution (in pixels). Examplesinclude digital pictures and fax attachments to an email, whichtypically can have resolutions in the 2000×2000 pixel range.

For wireless devices that support viewing of image attachments, thisrepresents a large amount of information for downloading, requiring alarge amount of bandwidth and associated cost thereof. Additionally, theuser must wait for an extended period of time while the device isdownloading the image.

Since mobile communication devices have limited screen real estate, thedownloaded image must be resized on the device in order to be viewable.However, such a resizing operation requires a very large memory and CPUusage thereby affecting the performance of all other applications on thedevice for the duration of the resizing process.

Companies such as Mapquest® allow users to zoom in on a map from a webbrowser client such that the map regenerates with a higher amount ofdetail. This zoom feature is accomplished in a web client/serverimplementation using vector graphics, which is not applicable to viewingwireless image attachments.

SUMMARY OF THE INVENTION

According to an aspect of the invention, a method is provided forallowing a user to quickly view an image attachment downloaded from aserver to a mobile communication device. Unlike the prior art systemsdiscussed above, the method of the invention relates to wireless imageattachment viewing, and the server side processing is performed onbitmap data and not vectors.

When a user issues a request to view a large image such as a fax ordigital photo image, the server resizes the image to match therequesting device display resolution. For example, if the user requestsa large fax image (e.g. 2400×1800 pixel resolution), the client device(i.e. mobile communication device) informs the server of its maximumresolution (e.g. 240×240 pixels), in response to which the serverresizes the image and returns the reduced resolution image to the devicefor viewing. This significantly reduces the amount of data downloadedwirelessly to the device as well as device memory consumed and CPU usagerequired to display the image.

Additionally, the server adjusts the number of colors in the image tomatch the color display capabilities of the requesting device. Forexample, only monochrome images will be returned to a monochrome deviceeven if the requested image is in color, thereby further reducing theamount of data downloaded wirelessly to the device.

One consequence of the server image resizing operation is decreasedlegibility of textual information and image details when viewed on thedevice, due to the reduced image resolution. Therefore, according to anadditional aspect of the invention, the client includes an “EnlargeArea” feature by which the user may zoom in on an area of detail of theresized image being viewed on the device, and request the server toprovide higher resolution of the enlarged detail. The “Enlarge Area”operation is performed in an iterative fashion making it possible tozoom in on a returned area of detail previously requested and therebyobtain an even higher area of detail.

Additional aspects and advantages will be apparent to a person ofordinary skill in the art, residing in the details of construction andoperation as more fully hereinafter described and claimed, referencebeing had to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of the preferred embodiment is set forth indetail below, with reference to the following drawings, in which:

FIG. 1 is a block diagram of a network environment in which thepreferred embodiment may be practiced;

FIG. 2 is a screen display of a menu for selecting the enlarge areacommand according to the preferred embodiment;

FIG. 3 shows image crop rectangle coordinates established at the deviceand sent to the server;

FIG. 4 is a flowchart showing an “Enlarge Area” request function on thedevice, according to the preferred embodiment;

FIG. 5 is a tree diagram showing the basic structure of a DocumentObject Model (“DOM”) used in the preferred embodiment;

FIG. 6 shows the top-level of the DOM structure in FIG. 5;

FIG. 7 shows an exemplary DOM structure for a word processing document;

FIG. 8 shows an exemplary DOM structure for a table document;

FIG. 9A shows a sample word processing document containing an imagesubdocument, and FIG. 9B shows an exemplary DOM structure therefore; and

FIG. 10 is a flowchart showing enlarge area image processing on theserver, according to the preferred embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference to FIG. 1, network environment 10 is shown in which thepreferred embodiment may be practiced. Network environment 10 includesat least one mobile communication device 12 communicating via a wirelessnetwork/Internet 14 to a server 28, via a firewall security server 29,for downloading document attachments to the devices 12. While only oneserver 28 is shown for illustration purposes, a person of skill in theart will understand that network environment 10 could have many suchservers for hosting web sites or graphic download sites, providingaccess to picture files such as JPEG, TIFF, BMP, PNG, SGI, MP4, MOV,GIF, SVG, etc. As would be understood by one of ordinary skill in theart, wireless networks 14 include GSM/GPRS, CDPD, TDMA, iDEN Mobitex,DataTAC networks, or future networks such as EDGE or UMTS, and broadbandnetworks like Bluetooth and variants of 802.11.

As discussed above, when the user of a mobile communication device 12views an image attachment using an “Attachment Viewer” application, aserver side resized version of the original image is downloaded to thedevice unless the original image has a smaller width and height than therequesting device screen size. The server 28 resizes the image to beviewed based on the requesting device screen width and height (inpixels). When the server 28 returns the resized image back to the device12 for viewing, the dimensions (width×height) of the original image arealso returned to the device. The original dimensions are required forcalculating the cropping rectangle for the “Enlarge Area” command, asdiscussed in greater detail below.

Specifically, while viewing an image in the Attachment Viewer the usercan select to zoom in on an image, which effectively magnifies theexisting pixels that constitute the image on the device 12. The “EnlargeArea” command causes the server 28 to generate a new image based on thecoordinates from the zoomed in image on the device 12 and returns thenew image to the device 12 for display.

With reference to FIG. 2, the “Enlarge Area” menu item is displayed inthe device menu if the following conditions are met: (A) there is nopending “Enlarge Area” request in process for the currently viewed imageand (B) the crop rectangle based on the currently zoomed image area onthe device is calculated to equate to the original image coordinates asit resides on the server in pixels and the calculated crop rectanglewidth is greater than the device screen width or the calculated croprectangle height is bigger than the device screen height.

Provided the foregoing conditions are met, the Attachment Viewerapplication on device 12 issues a request to the server 28 for theEnlarge Area operation to be performed on the original image stored onthe server.

When sending the request, the Attachment Viewer application includes acropping rectangle for the original image stored on the server 28, whichcorresponds to the currently viewed zoomed image area on the device 12.Thus, with reference to FIG. 2, the cropping rectangle is the viewablearea (left-hand side) that the user wishes to see returned from theserver 28 with higher quality/resolution (right-hand side).

The crop rectangle coordinates are the zoomed area coordinates on thedevice 12 calculated so they translate to the equivalent crop rectanglefor the original image stored on the server 28, as shown in FIG. 3.

The crop rectangle calculation also correctly allows for device sideoperations such as “Rotate” of the device side image prior to the“Enlarge Area” command being issued. The Attachment Viewer also sendsthe device screen characteristics of width, height and depth along withthe calculated crop rectangle for the original image when sending theEnlarge Area command to the server 28. The Attachment Server applicationthen returns the new image resulting from the enlarge area command formatching these device characteristics.

FIG. 4 shows the device side “Enlarge Area” command request procedure ofthe Attachment Viewer application running on device 12. At step 30, theAttachment Viewer displays an image attachment downloaded from theserver 28. As discussed above, the image is re-sized in the server ifthe dimensions exceed the screen size of the viewing device 12. Also, asdiscussed above, the server 28 returns the original image size to thedevice for calculating the crop rectangle.

Next, at step 32, the application determines whether precondition (A),above, is met for generating the “Enlarge Area” menu item (i.e. whetherthere is any pending Enlarge Area request in progress).

The crop rectangle coordinates are then calculated based on the imagezoom area defined on the device and translated using the original imagedimensions to crop rectangle coordinates for the original image storedon the server 28 (step 34).

Condition (B), discussed above, is then tested at step 36 and, providedthat the image width or height exceeds the device display width orheight, respectively, then the “Enlarge Area” menu is displayed (step37), and a determination is made as to whether the “Enlarge Area”command has been invoked (step 38).

At step 40, the device sends a request to the server 28 to enlarge thedisplayed image based on the translated crop rectangle coordinates andthe device screen dimensions.

If the enlarged image is received from the server 28 (a YES at decisionblock 41), then process control returns to step 30. Otherwise, an errorcode is displayed (step 42) and the process terminates (step 43).

Before turning to the server-side application for image areaenlargement, a description is provided herein of the structure of aDocument Object Model (DOM) for a document attachment to be viewed onthe mobile communication device 12.

The attachment server 28 uses a file-parsing distiller in the preferredembodiment, for a specific document type, to build an in-memory DocumentObject Model (“DOM”) structure representing an attachment of thatdocument type. The document DOM structure is stored in a memory cache ofserver 28, and can be iterated bi-directionally.

As shown in FIG. 5, the graph-based document DOM structure consists ofnodes and leaves. The nodes serve as the parents of leaves and nodes,while leaves are end points of a branch in the graph. Each node and leafcan have a set of attributes to specify its own characteristics. Forexample, a paragraph node can contain attributes to specify itsalignment, style, entry of document TOC, etc. In addition, each of thenodes and the leaves has a unique identifier, called a DOM ID, toidentify itself in the document DOM structure.

The document DOM structure is divided into three parts: top-level,component and references. The top level refers to the document rootstructure, while the main document is constructed in the component andthe references represent document references to either internal orexternal sub-document parts. The following paragraphs examine each partin detail.

The root node of a document DOM structure, referred to as “Document”,contains several children nodes, referred to as “Contents”, whichrepresent different aspects of the document contents. Each “Contents”node contains one or multiple “Container” nodes used to store variousdocument global attributes. The children of the “Container” nodes arecomponents, which store the document structural and navigationalinformation. When the attachment server 28 builds the

DOM structure for an attachment file for the first time, the top-levelstructure is a single parent-child chain as shown in FIG. 6.

Three types of components are defined by the attachment server 28: textcomponents, table components and image components, which represent text,tables, and images in a document, respectively. The text and tablecomponents are described in detail below, and the image componentstructure is identical.

A component consists of a hierarchy of command nodes. Each commandrepresents a physical entity, a property, or a reference defined in adocument. For the text component, the physical entity commands are page,section, paragraph, text segments, comments, footnote and endnotecommands which by name define the corresponding entity contained in adocument. The property commands for the text component are font, textcolor, text background color, hyperlink start/end and bookmark commands.The text component has only one reference command, referred to as thetext reference command, which is used to reference a subdocument definedin the main body of a document. Usually, the children of a textcomponent are page or section command nodes that, in turn, comprise aset of paragraph command nodes. The paragraph command can contain one ormultiple nodes for the remaining command types.

Using the following sample text document, the corresponding document DOMstructure is shown in FIG. 7:

First paragraph. Second paragraph with bold and red text.

As FIG. 7 demonstrates, the section command, which is the child of thetext component, consists of two paragraph commands. The first paragraphcommand contains one text segment command and the text content for thatparagraph is added as an attribute to the text segment command. Thesecond paragraph command has a relatively more complex structure, as thetext properties in the paragraph are much richer. Each time a textproperty (font, text color, etc.) changes, a corresponding text propertycommand is created and the change value is added to that command as anattribute. The subsequent text segment command records the text with thesame text property as an attribute. As document structure gets richerand more complex, more commands of corresponding types are created andthe document properties are added as attributes to those commands.

The table component has the same three types of commands as the textcomponent, but different command names. The document DOM structure forthe sample table document below is shown in FIG. 8:

Cell One Cell Two Cell Three Cell Four

As shown in the FIG. 8, the table component has physical entity typecommands of table, TableRow and TableCell, where the TableCell commandcan contain all available commands for the text component. In theexample above, the first child TableRow command of the table command hasan attribute “Index” defined by value of O. This indicates that theindicated table row is the first one defined in the table. The attributeof the leftmost table cell command in FIG. 8 has the same meaning.

A document sometimes contains subdocuments, for example images, tables,text boxes etc. The DOM structure set forth herein uses a referencecommand to point to the graph of such subdocuments. Thus, for the sampledocument of FIG. 9A, the attachment server 28 generates the DOMstructure shown in FIG. 9B.

The structure shown in FIG. 9 is identical to that discussed above inconnection with FIGS. 7 and 8, except for the attributes of the tworeference commands. The attachment server 28 constructs the image in“Sample Three” as a separate image component, which contains all of theimage data in its own DOM hierarchy. In the DOM structure for the maindocument, the values of the “Ref” attributes of those two referencecommands point to the image component, as indicated by the dashed lines,such that the DOM structure connects together all parts of the document.

Returning now to the preferred embodiment with reference to FIG. 10,when the attachment server 28 receives an “Enlarge Area” image requestfor an attachment (step 44), it first extracts the screen sizeinformation for the mobile communication device 12 and the croprectangle parameters from the request (step 46). After retrieving thedocument DOM structure for the attachment from the in-memory documentDOM cache (step 48), the server 28 traverses through the document DOMstructure to find out the corresponding image component in the DOMstructure for that image (step 50). The server 28 then iterates throughthe attributes of the image component to determine if a separate imagecomponent specifically for that screen size has already been constructed(step 52).

This separate image component contains the same graphic information asthe original image but is usually of smaller size (equal to or less thanthe device screen size), thus consuming much less device memory (usuallyin ratio of 1:15 to 1:20). If the separate image component does notexist, the attachment server 28 collects the image binary data from theoriginal image component in memory (step 54) and constructs the newimage component from the binary data (step 56).

At step 58, the server 28 caches the DSS image component as an attributeof the original image component in the DOM structure. These two imagecomponents contain essentially the same graphic information except thatthe original image component has a much higher resolution. A personskilled in the art will appreciate that this solution will consumeappreciable memory if there are a lot of different mobile device DSSdefinitions. Therefore, as an alternative, it would be possible toconstruct the DSS image component each time the server receives an“Enlarge Area” request, without caching it. However, based on thepremise that time is a more critical factor in wireless operation thanthe hardware deployment cost, caching the DSS image component isemployed in the preferred embodiment.

Similar to the operation of the separate image component construction,the attachment server 28 creates a new image data segment in memory bytailoring the original image binary data based on the crop rectangleparameters (step 60), and resizes the new image data segment based onthe device screen size to create the final image data segment (step 62).

In summary, the “Enlarge Area” device/server feature allows users toquickly retrieve any relevant part of a large image attachment that hasbeen resized by the server. This minimizes bandwidth usage, devicememory/CPU consumption, and request/response latency while stillallowing the user to view an image area in its original (native) levelof detail.

A person skilled in the art, having read this description of thepreferred embodiment, may conceive of variations and alternativeembodiments.

All such variations and alternative embodiments are believed to bewithin the ambit of the claims appended hereto.

What is claimed is:
 1. A method comprising: transmitting, from a serverto a device, original size data for an image; receiving a request at theserver from the device to enlarge an area of the image, the imagepreviously transmitted to the device from the server, the requestcomprising translated crop rectangle coordinates derived from the areato be enlarged and the original size data; building a graph structurerepresenting a map of the image; constructing a new image component frombinary data collected from the image; caching the new image component asan attribute of the image in the graph structure; modifying the binarydata based on the translated crop rectangle coordinates and an imagesize limit associated with the device to create an enlarged area of theimage; and transmitting an enlarged area of the image for display. 2.The method of claim 1, wherein the graph structure is a Document ObjectModel (DOM).
 3. The method of claim 1, further comprising: prior to thereceiving the request to enlarge the area of the image, receiving arequest from the device for the image, the request including the imagesize limit; and thereafter transmitting the image to the mobilecommunication device, the image being re-sized in the event that thedimensions thereof exceed the image size limit.
 4. The method of claim3, wherein the image size limit is indicative of at least one of screensize and display resolution of the mobile communication device.
 5. Themethod of claim 1, further comprising: traversing the graph structure tolocate any corresponding image component for the original image and,upon locating the corresponding image component, determining if aseparate image component has been constructed for the image size limit;and the constructing the new image component from the binary data occurswhen the separate image component has not been constructed and theconstructing comprising collecting initial image binary data from theimage.
 6. A method comprising: transmitting, from a server to a device,original size data for an image, when there is not a pending enlargearea request; receiving a request at the server from the device toenlarge an area of the image, the request comprising translated croprectangle coordinates derived from the area to be enlarged and theoriginal size data; building a graph structure representing a map of theimage; traversing the graph structure to locate any corresponding imagecomponent for the image and, upon locating the corresponding imagecomponent iterating through attributes of the image component todetermine if a separate image component has been constructed for animage size limit associated with the device; when the separate imagecomponent has not been constructed, collecting initial image binary datafrom the image and constructing a new image component from the binarydata; caching the new image component as an attribute of the image inthe graph structure; collecting and modifying the image binary data fromthe image based on the translated crop rectangle coordinates and theimage size limit to create an enlarged area of the image when: a widthof the image is greater than an image size width limit; or a height ofthe image is greater than image size height limit; and transmitting anenlarged area of the image for display.
 7. The method of claim 6,wherein the graph structure is a Document Object Model (DOM).
 8. Aserver comprising: a hardware element for transmitting original sizedata for an image, when there is not a pending enlarge area request; ahardware element for receiving a request to enlarge an area of theimage, the request comprising translated crop rectangle coordinatesderived from the area to be enlarged and the original size data; ahardware element for building a graph structure representing a map ofthe image; a hardware element for traversing the graph structure tolocate any corresponding image component for the image and, uponlocating the corresponding image component iterating through attributesof the image component to determine if a separate image component hasbeen constructed for an image size limit associated with the device; ahardware element for, when the separate image component has not beenconstructed, collecting initial image binary data from the image andconstructing a new image component from the binary data; a hardwareelement for caching the new image component as an attribute of the imagein the graph structure; a hardware element for collecting and modifyingthe image binary data from the image based on the translated croprectangle coordinates and the image size limit to create an enlargedarea of the image when: a width of the image is greater than an imagesize width limit; or a height of the image is greater than image sizeheight limit; and a hardware element for transmitting an enlarged areaof the image for display.